Surgical robotic system

ABSTRACT

A surgical robotic system is disclosed to include an operating table, a plurality of robotic arms and surgical instruments, a user console, and a control tower. The plurality of robotic arms are mounted on the operating table and can be stowed folded under the table for storage. The user console has one or more user interface devices, which function as master devices to control the plurality of surgical instruments.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Patent Application No.62/754,869, filed Nov. 2, 2018, which is hereby incorporated byreference.

TECHNICAL FIELD

The subject technology generally relates to robotics and surgicalsystems, and more specifically to system architectures and components ofa surgical robotic system for minimally invasive surgeries.

BACKGROUND

Minimally-invasive surgery (MIS), such as laparoscopic surgery, involvestechniques intended to reduce tissue damage during a surgical procedure.For example, laparoscopic procedures typically involve creating a numberof small incisions in the patient (e.g., in the abdomen), andintroducing one or more surgical tools (e.g., end effectors andendoscope) through the incisions into the patient. The surgicalprocedures may then be performed using the introduced surgical tools,with the visualization aid provided by the endoscope.

Generally, MIS provides multiple benefits, such as reduced patientscarring, less patient pain, shorter patient recovery periods, and lowermedical treatment costs associated with patient recovery. Recenttechnology development allows more MIS to be performed with roboticsystems that include one or more robotic arms for manipulating surgicaltools based on commands from a remote operator. A robotic arm may, forexample, support at its distal end various devices such as surgical endeffectors, imaging devices, cannulas for providing access to thepatient's body cavity and organs, etc. In robotic MIS systems, it may bedesirable to establish and maintain high positional accuracy forsurgical instruments supported by the robotic arms.

Existing robotically-assisted surgical systems usually consist of asurgeon console that resides in the same operating room as the patientand a patient-side cart with four interactive robotic arms controlledfrom the console. Three of the arms hold instruments such as scalpels,scissors, or graspers, while the fourth arm supports an endoscopecamera. In order to reposition the patient during a surgical procedure,surgical staff may have to undock the instruments/arms, reposition thearms/patient cart, and re-dock the instruments/arms. It may be desirableto have a surgical robotic system that allows repositioning the patientwithout undocking arms during general or other laparoscopic surgeries.

SUMMARY

Disclosed herein is a robotically-assisted surgical system, which is asoftware-controlled, electro-mechanical system, designed for surgeons toperform minimally-invasive surgery. The surgical robotic system may becomprised of three major sub-systems: a surgeon subsystem—the userconsole (surgeon console or surgeon bridge), a central controlsubsystem—the control tower, and a patient subsystem—the table androbotic arms. A surgeon seated in a surgeon seat of the user console maycontrol the movement of compatible instruments using master user inputdevices (UIDs) and foot pedals. The surgeon can view a three-dimensional(3D) endoscopic image on a high-resolution open stereo display, whichprovides the surgeon the view of the patient anatomy and instrumentationalong with icons, apps, and other user interface features. The userconsole may also provide an option for immersive display using aperiscope, which can be pulled from the back of the surgeon seat.

The control tower can function as the control and communication centerof the surgical robotic system. It may be a mobile point-of-care carthousing a touchscreen display, and include computers that control thesurgeon's robotically-assisted manipulation of instruments, safetysystems, a graphical user interface (GUI), an advanced light engine(also referred to as a light source), and video and graphics processors,among other supporting electronic and processing equipment. The controltower can also house third-party devices like an electrosurgicalgenerator unit (ESU), and insufflator and CO₂ tanks.

The patient subsystem may be an articulated operating room (OR) tablewith up to four integrated robotic arms positioned over the targetpatient anatomy. The robotic arms of the surgical system may incorporatea remote center design, i.e., each arm pivots about a fixed point inspace where the cannula passes through a patient's body wall. Thisreduces lateral movement of the cannula and minimizes stresses at thepatient's body wall. A suite of compatible tools can beattached/detached from an instrument driver mounted to the distal end ofeach arm, enabling the surgeon to perform various surgical tasks. Theinstrument drivers can provide intracorporeal access to the surgicalsite, mechanical actuation of compatible tools through a sterileinterface, and communication with compatible tools through a sterileinterface and user touchpoints. An endoscope can be attached to any armand provide the surgeon with the high resolution, three-dimensional viewof the patient anatomy. The endoscope can also be used endoscopically(hand-held) at the start of a surgery and then be mounted on any one ofthe four arms. Additional accessories such as trocars (also calledsleeves, seal cartridge, and obturators) and drapes may be needed toperform procedures with the surgical robotic system.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example operating room environmentwith a surgical robotic system, in accordance with aspects of thesubject technology.

FIG. 2A is a schematic diagram illustrating one exemplary design of arobotic arm, a tool drive, and a cannula loaded with a robotic surgicaltool; and FIGS. 2B and 2C are schematic diagrams illustrating anexemplary tool drive with and without a loaded tool adjacent,respectively, in accordance with aspects of the subject technology.

FIG. 3A is a side view of a surgical table, a table adapter, and tworobotic arms coupled to the table adapter and in a stowed position; andFIG. 3B is a side view of the surgical table, adapter, and arms shown inan operating position with three arms positioned on one side of thetable, in accordance with aspects of the subject technology.

FIG. 4A is a schematic diagram illustrating a rear perspective view ofan exemplary user console; and FIG. 4B is a schematic illustration ofadjustable settings or parameters of the exemplary user console, inaccordance with aspects of the subject technology.

FIG. 5A is an illustrative schematic of a variation of a wired handhelduser interface device in an egg shape; FIG. 5B is an illustrativeschematic of a variation of a wireless handheld user interface devicewith modular and interchangeable adapters; FIG. 5C is an illustrativeschematic of a variation of a wired handheld user interface device in acylinder shape; and FIG. 5D is an illustrative schematic of a variationof a handheld user interface device with grip linkages, in accordancewith aspects of the subject technology.

FIG. 6A is a screenshot illustrating an exemplary variation of a GUIwith tool notifications; and FIG. 6B is a screenshot illustratinganother exemplary variation of a GUI with sidebar panels, in accordancewith aspects of the subject technology.

FIG. 7 is a schematic diagram illustrating an exemplary control tower ofa surgical robotic system, in accordance with aspects of the subjecttechnology.

FIG. 8 is a block diagram illustrating exemplary hardware components ofa surgical robotic system, in accordance with aspects of the subjecttechnology.

DETAILED DESCRIPTION

Examples of various aspects and variations of the subject technology aredescribed herein and illustrated in the accompanying drawings. Thefollowing description is not intended to limit the invention to theseembodiments, but rather to enable a person skilled in the art to makeand use this invention.

System Overview

A robotic-assisted surgical system disclosed herein is asoftware-controlled, electro-mechanical system designed for surgeons toperform minimally-invasive surgery. The surgical robotic system can beused with an endoscope, compatible endoscopic instruments, andaccessories. The system may be used by trained physicians in anoperating room environment to assist in the accurate control ofcompatible endoscopic instruments during robotically-assisted urologic,gynecologic and other laparoscopic surgical procedures. The system alsoallows the surgical staff to reposition the patient by adjusting thetable without undocking the robotic arms during urologic, gynecologicand other laparoscopic surgical procedures. The compatible endoscopicinstruments and accessories for use with the surgical system areintended for endoscopic manipulation of tissue including grasping,cutting, blunt and sharp dissection, approximation, ligation,electrocautery, and suturing.

FIG. 1 is a diagram illustrating an example operating room environmentwith a surgical robotic system 100, in accordance with aspects of thesubject technology. As shown in FIG. 1 , the surgical robotic system 100comprises a user console 110, a control tower 130, and a surgical robot120 having one or more surgical robotic arms 122 mounted on a surgicalplatform 124 (e.g., a table or a bed etc.), where surgical tools withend effectors are attached to the distal ends of the robotic arms 122for executing a surgical procedure. The robotic arms 122 are shown astable-mounted, but in other configurations, the robotic arms may bemounted in a cart, a ceiling, a sidewall, or other suitable supportsurfaces.

Generally, a user, such as a surgeon or other operator, may be seated atthe user console 110 to remotely manipulate the robotic arms 122 and/orsurgical instruments (e.g., teleoperation). The user console 110 may belocated in the same operation room as the robotic system 100, as shownin FIG. 1 . In other environments, the user console 110 may be locatedin an adjacent or nearby room, or tele-operated from a remote locationin a different building, city, or country. The user console 110 maycomprise a seat 112, pedals 114, one or more handheld user interfacedevices (UIDs) 116, and an open display 118 configured to display, forexample, a view of the surgical site inside a patient. As shown in theexemplary user console 110, a surgeon siting in the seat 112 and viewingthe open display 118 may manipulate the pedals 114 and/or handheld userinterface devices 116 to remotely control the robotic arms 122 and/orsurgical instruments mounted to the distal ends of the arms 122.

In some variations, a user may also operate the surgical robotic system100 in an “over the bed” (OTB) mode, in which the user is at thepatient's side and simultaneously manipulating a robotically-driventool/end effector attached thereto (e.g., with a handheld user interfacedevice 116 held in one hand) and a manual laparoscopic tool. Forexample, the user's left hand may be manipulating a handheld userinterface device 116 to control a robotic surgical component, while theuser's right hand may be manipulating a manual laparoscopic tool. Thus,in these variations, the user may perform both robotic-assisted MIS andmanual laparoscopic surgery on a patient.

During an exemplary procedure or surgery, the patient is prepped anddraped in a sterile fashion to achieve anesthesia. Initial access to thesurgical site may be performed manually with the robotic system 100 in astowed configuration or withdrawn configuration to facilitate access tothe surgical site. Once the access is completed, initial positioningand/or preparation of the robotic system may be performed. During theprocedure, a surgeon in the user console 110 may utilize the pedals 114and/or user interface devices 116 to manipulate various end effectorsand/or imaging systems to perform the surgery. Manual assistance mayalso be provided at the procedure table by sterile-gowned personnel, whomay perform tasks including but not limited to, retracting tissues orperforming manual repositioning or tool exchange involving one or morerobotic arms 122. Non-sterile personnel may also be present to assistthe surgeon at the user console 110. When the procedure or surgery iscompleted, the robotic system 100 and/or user console 110 may beconfigured or set in a state to facilitate one or more post-operativeprocedures, including but not limited to, robotic system 100 cleaningand/or sterilization, and/or healthcare record entry or printout,whether electronic or hard copy, such as via the user console 110.

In some aspects, the communication between the surgical robot 120 andthe user console 110 may be through the control tower 130, which maytranslate user input from the user console 110 to robotic controlcommands and transmit the control commands to the surgical robot 120.The control tower 130 may also transmit status and feedback from therobot 120 back to the user console 110. The connections between thesurgical robot 120, the user console 110 and the control tower 130 maybe via wired and/or wireless connections, and may be proprietary and/orperformed using any of a variety of data communication protocols. Anywired connections may be optionally built into the floor and/or walls orceiling of the operating room. The surgical robotic system 100 mayprovide video output to one or more displays, including displays withinthe operating room, as well as remote displays accessible via theInternet or other networks. The video output or feed may also beencrypted to ensure privacy and all or portions of the video output maybe saved to a server or electronic healthcare record system.

Prior to initiating surgery with the surgical robotic system, thesurgical team can perform the preoperative setup. During thepreoperative setup, the main components of the surgical robotic system(table 124 and robotic arms 122, control tower 130, and user console110) are positioned in the operating room, connected, and powered on.The table 124 and robotic arms 122 may be in a fully-stowedconfiguration with the arms 122 under the table 124 for storage and/ortransportation purposes. The surgical team can extend the arms fromtheir stowed position for sterile draping. After draping, the arms 122can be partially retracted until needed for use. A number ofconventional laparoscopic steps may need to be performed includingtrocar placement and insufflation. For example, each sleeve can beinserted with the aid of an obturator, into a small incision and throughthe body wall. The sleeve and obturator allow optical entry forvisualization of tissue layers during insertion to minimize risk ofinjury during placement. The endoscope is typically placed first toprovide hand-held camera visualization for placement of other trocars.After insufflation, if required, manual instruments can be insertedthrough the sleeve to perform any laparoscopic steps by hand.

Next, the surgical team may positions the robotic arms 122 over thepatient and attach each arm 122 to its corresponding sleeve. Thesurgical robotic system 100 has the capability to uniquely identify eachtool (endoscope and surgical instruments) as soon as it is attached anddisplay the tool type and arm location on the open or immersive display118 at the user console 110 and the touchscreen display on the controltower 130. The corresponding tool functions are enabled and can beactivated using the master UIDs 116 and foot pedals 114. Thepatient-side assistant can attach and detach the tools, as required,throughout the procedure. The surgeon seated at the user console 110 canbegin to perform surgery using the tools controlled by two master UIDs116 and foot pedals 114. The system translates the surgeon's hand,wrist, and finger movements through the master UIDs 116 into precisereal-time movements of the surgical tools. Therefore, the systemconstantly monitors every surgical maneuver of the surgeon and pausesinstrument movement if the system is unable to precisely mirror thesurgeon's hand motions. In case the endoscope is moved from one arm toanother during surgery, the system can adjust the master UIDs 116 forinstrument alignment and continue instrument control and motion. Thefoot pedals 114 may be used to activate various system modes, such asendoscope control and various instrument functions including monopolarand bipolar cautery, without involving surgeon's hands removed from themaster UIDs 116.

The table 124 can be repositioned intraoperatively. For safety reason,all tool tips should to be in view and under active control by thesurgeon at the user console 110. Instruments that are not under activesurgeon control must be removed, and the table feet must be locked.During table motion, the integrated robotic arms 122 may passivelyfollow the table movements. Audio and visual cues can be used to guidethe surgery team during table motion. Audio cues may include tones andvoice prompts. Visual messaging on the displays at the user console 110and control tower 130 can inform the surgical team of the table motionstatus.

Arms and Table

FIG. 2A is a schematic diagram illustrating one exemplary design of arobotic arm 112, a tool drive 220, and a cannula 221 loaded with arobotic surgical tool 250, in accordance with aspects of the subjecttechnology. As shown in FIG. 2A, the example surgical robotic arm 112may include a plurality of links (e.g., links 201-208A-B) and aplurality of actuated joint modules (e.g., joints 211-217) for actuatingthe plurality of links relative to one another. The joint modules mayinclude various types, such as a pitch joint or a roll joint, which maysubstantially constrain the movement of the adjacent links aroundcertain axes relative to others. Also shown in the exemplary design ofFIG. 2A is a tool drive 220 attached to the distal end of the roboticarm 112. The tool drive 220 may include a cannula 221 coupled to its endto receive and guide a surgical instrument 250 (e.g., endoscopes,staplers, etc.). The surgical instrument (or “tool”) 250 may include anend effector having a robotic wrist 252 and jaws 254 at the distal endof the tool. The plurality of the joint modules of the robotic arm 112can be actuated to position and orient the tool drive 220, whichactuates the robotic wrist 252 and the end effector 254 for roboticsurgeries.

In some variations, the plurality of links and joints of the robotic arm112 can be divided into two segments. The first segment (setup arm)includes links 201-205 and joints 211-215 that provide at least fivedegrees of freedom (DOFs). The proximal end of the first segment can bemounted to a fixture, while the distal end is coupled to the secondsegment. The second segment (spherical arm) includes links 206-208providing the arm with at least two DOFs. Link 208 may comprise a firstlink 208A and a second link 208B operatively coupled with a pulleymechanism to form a parallelogram and to constrain the movement of thetool drive 220 around a mechanical remote center of motion (RCM). Thefirst segment may be referred to as the setup arm because it mayposition and adjust the RCM in space relative to the mounting fixture,while the second segment may be referred to as the spherical arm becauseit is configured to move the surgical tool within a generally sphericalworkspace.

FIGS. 2B and 2C are schematic diagrams illustrating an exemplary tooldrive with and without a loaded tool adjacent, respectively, inaccordance with aspects of the subject technology. As shown in FIG. 2B,in one variation, the tool drive 220 may include an elongated base (or“stage”) 222 having longitudinal tracks 223 and a tool carriage 224,which is slidingly engaged with the longitudinal tracks 223. The stage222 may be configured to couple to the distal end of a robotic arm suchthat articulation of the robotic arm positions and/or orients the tooldrive 220 in space. Additionally, the tool carriage 224 may beconfigured to receive a tool base 252 of the tool 220, which may alsoinclude a tool shaft 254 extending from the tool base 252 and throughthe cannula 221, with the robotic wrist 252 and the end effector 254 (asshown in FIG. 2A) disposed at the distal end.

Additionally, the tool carriage 224 may actuate a set of articulatedmovements of the robotic wrist and the end effector, such as through acable system or wires manipulated and controlled by actuated drives (theterms “cable” and “wire” are used interchangeably throughout thisapplication). The tool carriage 224 may include different configurationsof actuated drives. For example, the rotary axis drives may include amotor with a hollow rotor and a planetary gear transmission at leastpartially disposed within the hollow rotor. The plurality of rotary axisdrives may be arranged in any suitable manner. For example, the toolcarriage 224 may include six rotary drives 226A-226F arranged in tworows, extending longitudinally along the base that are slightlystaggered to reduce width of the carriage and increase the compactnature of the tool drive. As clearly shown in FIG. 2C, rotary drives226A, 226B, and 226C may be generally arranged in a first row, whilerotary drives 226D, 226E, and 226F may be generally arranged in a secondrow that is slightly longitudinally offset from the first row.

FIG. 3A is a side view of a surgical table 300, a table adapter 328 andtwo robotic arms 330A-B coupled to the table adapter and in a stowedposition, in accordance with aspects of the technology. As shown in FIG.3A, a surgical table 300 includes a table top 320, a pedestal 322 (alsoreferred to herein as support), and a base 324. The pedestal 322 can bemounted to the base 324, which can be fixed to the floor of an operatingroom, or can be movable relative to the floor. The table top 320includes a head section 315, a torso section 317, and a leg section 319along a y-axis in the longitude direction. The table top 320 can alsoinclude an arm section(s) (not shown). The table top 320 has a topsurface on which a patient can be disposed. The pedestal 322 can providefor movement of the table top 320 in a desired number of degrees offreedom. For example, in one embodiment, the head section 316 ispivotally coupled to a first end of the torso section 317, and the legsection 319 is pivotally coupled to a second end of the torso section317, such that the head section 316 and leg section 319 can be rotatedabout an x-axis extending in a lateral direction relative to the tabletop 320. The movement of the table top 320 and/or its constituentsections may be performed manually or driven by motors, controlledlocally or remotely.

As shown in FIG. 3A, a table adapter 328 (also referred to herein as“adapter”) including a table interface structure 340 is coupled to thepedestal 322 of the surgical table 300. The table interface structure340 can be a single structure coupled to the pedestal 322 and/or thetable top 320 that supports one or more robotic arms 330A-B. Via thetable adapter 328, arm 330A and arm 330B may be mounted on the same side(e.g., the side shown in FIG. 3A) relative to the longitude direction ofthe table top 320. The adapter 328 may also include two interfacemechanisms 340 coupled to the table 300 on opposite sides of the tabletop 320. For example, two more arms may be mounted on the other side(not shown) of the table top 320. When not in use, arm 330A (and oneother arm on the opposite side not shown) may be stowed under the headsection 315, while arm 330B (and another arm on the opposite side notshown) may be stowed under the leg section 319.

In some variations, the adapter 328 can be coupled to the pedestal 322,such that the adapter 328 can move vertically up and down relative tothe pedestal 322. Additionally, or in alternative configurations, thetable top 320 and the adapter 328 can be moved relative to each otherwithin the plane of the table top 320. The adapter 328 may furtherinclude multiple first link members 332 that are each pivotally coupledto the table interface structure 340 at a first joint 333, and multiplesecond link members 334 that are each coupled to one of the first linkmembers 332. A second link member 334 may be allowed to translatevertically relative to a corresponding first link member 332. In somevariations, the second link member 334 can include a coupling portionconfigured to receive a mounting base of the robotic arm 330A-B. In theexample shown in FIG. 3A, the coupling portion also includes the firstjoint 335 (J1) of the robotic arm 330B, which can be folded or collapsedunder the table 300 for storage.

FIG. 3B is a side view of the surgical table 300, adapter 328, and arms330A-C shown in an operating position with three arms 330A-C on one sideof the table 300. In this example, a second joint 335 (J1) is includedin the pivotal coupling between the first link member 332 and the secondlink member 334, so that the adapter 328 and arms 330A-C can be movedfrom the stowed position to various operating positions by extending thearms 330A-C via pivot joints 333 (J0) and arm joints 335 (J1). As shownin FIG. 3B, three arms 330A-C are currently positioned on the same sideof the table 300. To achieve this configuration, one arm from the pairof arms coupled to the adapter 328 on one side of the table 300 (e.g.,the side not shown in FIG. 3B) can be pivoted via the pivot joint 333inwardly (e.g., coming out of paper) relative to the longitude axis ofthe table top 320 and extended to the other side of the table 300 (e.g.,the side shown in FIG. 3B). For example, from the stowed configuration,arm 330C can be pivoted under the table top 320 (and the opposite arm330A) across the longitude direction (i.e., y-axis) and extended upwardusing the second pivot joint 335 (J1) to the other side from under thetable top 320. Such a configuration may be used to perform, for example,a prostatectomy procedure.

In same variations, the table 300 may be articulated in six (or fewer)degrees of freedom including translations and rotation (tilting) alongX, Y, and Z axes, as shown in both FIGS. 3A and 3B. For example, theheight of the pedestal 322 can be adjusted, which together withlongitudinal or lateral motions (translation or rotation/tilting) of thetable top 320, can allow for the table top 320 to be positioned at adesired surgical site at a certain height above the floor (e.g., toallow surgeon access) and a certain distance from the support 322. Thetable adapters 328 may be fixed to the bottom of the table top 320 butabove any table motors (or joints) that enable the adjustments of thetable top 320 along the pedestal 322. As a result, when the table top320 is articulated, all the robotic arms 330A-C coupled to the table 300would automatically move with the table top 320, thus maintainingrelative poses (positions and orientations) during (and after)translation and rotation of the table top 320 without the need ofrepositioning.

User Console and UID

Generally, as shown in FIGS. 4A and 4B, a user console 110 for remotelycontrolling a remote surgical robotic instrument includes a base 402, anadjustable ergonomic seat assembly 410 comprising a seat pan 406, anopen display panel 450 configured to receive real-time surgicalinformation, and a pedal assembly 430. The seat assembly 410 may beselectively configurable in a plurality of user console configurations(e.g., seated, reclined, and/or elevated). In some variations, othercomponents of the user console 110 (e.g., display, controls, etc.) maysimilarly have multiple positions or configurations corresponding to theuser console configurations, where the other components automaticallyadjust their positions or configurations in response to a selected seatassembly configuration. Furthermore, one or more of the components ofthe user console 110 may automatically adjust their relative positionsand/or orientations according to a seating profile associated with atleast one user. Exemplary variations of a user console 110 are describedin further detail in U.S. patent application Ser. No. 15/712,052 titled“USER CONSOLE SYSTEM FOR ROBOTIC SURGERY” filed on Sep. 21, 2017, whichis incorporated herein in its entirety by this reference.

In some variations, the user console 110 may additionally oralternatively include an immersive display, such as a periscope 420 orother head-mounted display placed in contact with the user's face orhead, as shown in FIG. 4A. The immersive display 420 may be coupled tothe seat assembly 410 via an immersive display support arm 422 thatpositions the immersive display 420 in front of the face of a userlocated in the seat assembly 410, such that the user may directly viewcontent in the immersive display 410 in an immersive manner (e.g.,comfortably and ergonomically immerses the user into the displayenvironment with reduced distractions from the user's peripheral fieldof view). Both the open display 450 and the immersive display 420 maydisplay various information associated with the surgical procedure(e.g., endoscopic camera view of the surgical site, static images, GUIs,etc.) and/or robotic surgical system (e.g., status, system settings),and/or other suitable information in the form of 2D and 3D video,images, text, graphical interfaces, warnings, controls, indicatorlights, etc. In addition, both displays may track user's gazes, headgestures, and other head/eye movements to interact with displayedcontent and to control the operation of other instruments in the roboticsurgical system.

To facilitate the adjustment and set-up of the user console 110 forergonomic and other adjustments, a configuration controller may beprovided that can detect and save the specific configuration of one ormore components of the user console 110, including the seat assembly410, display assembly 450, pedal assembly 430, and also control themotors of the adjustment mechanisms to restore the user console 110 to asave specific configuration. Each configuration may be linked to one ormore users, user characteristics, patient or patient characteristics(e.g., height, weight), operating teams, robot system configurations,seating preferences, and/or surgery types. The configuration controllermay be separate from the robot controller of the robot system, with itsown processor, memory, and input/output interface to the motors,interlocks, actuators, and/or sensors, or may be part of the samesystem. In addition to any customized configuration, the configurationcontroller may include some pre-configured settings, or may include analgorithm that sets a configuration based upon the height and/or weightof the user as entered into the configuration controller, or measured byone or more sensors built into the user console (e.g., weight sensors inthe seat pan 406, base 402, and/or pedal assembly 430, optical height,or length detection).

FIG. 4B illustrates an exemplary set of parameters that may, in somevariations, be adjustable to configure the user console 110. The opendisplay may be adjustable in several degrees of freedom (“DOF”). Forexample, open display height may be adjusted via vertical translation(“ODV”) along a display support, anterior-posterior location of the opendisplay may be adjusted via horizontal translation (“ODH”) relative to abase of the user console, and open display tilt (“ODT”) may be adjusted(e.g., relative to the display support). Additionally or alternatively,the pedal tray 430 may be adjustable in up to three or more DOF. Forexample, foot pedal tray tilt (“PT”) may be adjusted (e.g., relative tothe base of the user console), anterior-posterior location of the footpedal tray may be adjusted via horizontal translation (“PH”) relative tothe base of the user console, and/or height of the foot pedal tray maybe adjusted via vertical translation (not shown) such as with anadjustable riser coupled to the base of the user console.

The seat assembly may furthermore be adjustable in a variety of DOFs.For example, seat rotational position may be adjusted via seat swivel(“CS”) around a vertical axis (e.g., by adjusting rotational swivelposition of a seat support pillar as described herein), seat height(“CV”) may be adjusted via translation along the seat support pillar,seat recline (generally shown as seat recline (“CR”)) relative to a seatpan may be adjusted (e.g., as described in further detail herein), andseat pan tilt (“CBT”) may be adjusted (e.g., as described in furtherdetail herein). The armrest height (“AV”) and other arm restconfigurations (e.g., lateral or planar motion, as described furtherherein) may be adjusted. Additionally, headrest height (“HV”) and/orheadrest tilt (“HT”) may be adjusted (e.g., as further describedherein). Furthermore, height of the base (“By”) relative to the groundmay be adjusted, for example, as the result of deployment of wheels (asfurther described below) for transport and other suitable purposes.

In use, the user console 110 may be adjusted to a desired configuration,including ergonomic adjustments to the seat assembly 410, pedal assembly430, and display assembly 450, but also customizations to the userinterface and user interface devices, if available. The complete or asubset of the configuration may then be saved, and optionally linked toone identifier or category. The identifiers may be a user identifier, anon-user identifier category or characteristic, (e.g., surgery type,seating arrangement, etc.) and/or a biometric identifier (e.g., iriscode, fingerprint, etc.). In subsequent usage, one or more identifiersare entered, provided, or selected simultaneously or serially, to narrowdown the saved configuration(s) for final selection or confirmation. Theconfiguration controller then signals or controls the various motors tomake any mechanical adjustments to the user console 110, and alsoreconfigures or sets the configuration of the user interface. This canoccur while the user is seated in the user console 110 or prior toseating, in order to reduce set-up time for a single user, or betweenmultiple users who use the same user console 110 in respectivecustomized configurations during a single procedure, etc. Furthermore,in some variations, the user console 110 may dynamically improveergonomics by tracking motions of the user (e.g., body position, eyeposition, eye gaze, etc.) and, in response to the user's motions,automatically recommending or transitioning to an optimum configurationfor the user's ergonomic, viewing, hardware, and/or other needs, such asfor reducing fatigue or injury. For example, if the configurationcontroller detects that the user in the seat assembly 410 begins tostrain upwards (as if trying to obtain a higher perspective of theoperating table), the controller may automatically adjust the seatassembly 210 to elevate the user.

The plurality of user console configurations may include seated,reclined, and elevated configurations. In a seated configuration, theseat pan 406 may be at a height wherein the user's heels are in contactwith or generally about the base 402 while the seat pan 406 and theuser's thighs are generally aligned. The pedal assembly 430 ispositioned at an anterior-posterior position wherein the user's forefootis in contact with the pedals 430 and angled about perpendicularly tothe user's lower leg. In an elevated configuration, the seat pan 406 canbe elevated relative to the position in the seated configuration. Theheight of the seat pan 406 may be set such that the heel of the user isin contact with the base 402, but in other variations may be configuredsuch that the user's heels are above and not in contact with the base402. In a reclined configuration, the seat pan 406 is in a retrovertedorientation along with the display panel 450.

The user interface at the user console for controlling a surgicalrobotic system may also include one or more handheld user interfacedevices (UIDs) mounted on the armrest of the seat assembly or below theopen display, among other possible locations. In some variations, a userinterface device held in the left hand of the user may be configured tocontrol an end effector represented on a left side of a camera viewprovided to the user, while a user interface device held in the righthand of the user may be configured to control an end effectorrepresented on a right side of the camera view. The control inputs tothe user interface device may, for example, be provided by the user asinput commands during the course of providing a diagnostic, surgical,laparoscopic, or minimally-invasive surgical procedure, or other roboticprocedure. Exemplary variations of a user interface device are describedin further detail in U.S. patent application Ser. No. 15/836,420 titled“USER INTERFACE DEVICES FOR USE IN ROBOTIC SURGERY” filed on Dec. 8,2017, which is incorporated herein in its entirety by this reference.

In some variations, the handheld user interface device may be agroundless or ungrounded user input device configured to be held in thehand and manipulated in free space. For example, the user interfacedevice may be configured to be held between the fingers of a user andmoved about freely (e.g., translated, rotated, tilted, etc.) by the useras the user moves his or her arms, hands, and/or fingers. Additionallyor alternatively, the handheld user interface device may be abody-grounded user interface device, in that the user interface devicemay be coupled to a portion of the user (e.g., to fingers, hand, and/orarms of a user) directly or via any suitable mechanism such as a glove,hand strap, sleeve, etc. Such a body-grounded user interface device maystill enable the user to manipulate the user interface device in freespace. Accordingly, in variations in which the user interface device isgroundless or body-grounded (as opposed to permanently mounted orgrounded to a fixed console or the like), the user interface device maybe ergonomic and provide dexterous control, such as by enabling the userto control the user interface device with natural body movementsunencumbered by the fixed nature of a grounded system.

FIGS. 5A-5D are illustrations of various designs for a handheld userinterface device (UID) 116, in accordance with aspects of the subjecttechnology. The UID 116 may include a housing 520 and one or morecircumferential or partially-circumferential lips (or raised rings) 525.In the wired variations shown in FIGS. 5A and 5C, the UID 116 alsoincludes a wire 550 coupling the UID 116 to the user console. In somevariations, the housing 520 may have a lumen or other internal volumeconfigured to receive electronics and/or other components. For example,the internal volume may include at least one printed circuit board (PCB)and a battery for powering the PCB and other electrical components inthe UID 116. Furthermore, one or more sensors, such as accelerometers,gyroscopes, magnetometers, and/or other optical, magnetic, or capacitivesensors (not shown) can be disposed inside or on an outer surface of thehousing 520 for tracking positions and orientations of the UID 116 insix degree of freedom. The UID 116 may further include one or moregesture detection sensors, for example, a proximity sensor or a grip orsqueeze sensor configured to detect deformation of the housing 520,where the detected deformation of the housing 520 may be mapped to acontrol of the graphic user interface or the robotic system.

In some variations, as shown in FIGS. 5A-5D, the housing 520 of the UID116 may have a first end 530A (e.g., proximal end) and a second end 530B(e.g., distal end), where the first end 530A and/or the second end 530Bincludes an engagement feature configured to couple to a detachableadapter. The detachable adapter may be interchangeable with other kindsof detachable adapters, thereby facilitating a modular design permittingmultiple configurations of user interface devices, such as withdifferent form factors, different functional features, and/or differenttracking technologies in various combinations. For example, FIG. 5Bdepicts a disk adapter 530B that can be removably engaged onto thehousing 520 and function as a joystick. In FIG. 5C, a camera adapter530B can be mounted on the distal end of the UID 116 for opticaltracking with the wire 550 coupled to the side of the UID housing 520.As another example, FIG. 5D shows another variation of the UID 116 thatinclude a proximal end 530A coupled to or integrated with aflower-shaped grip linkage. The grip linkage may have a plurality ofgrip components 502, each includes a grip crank 504 and a follower arm506 pivotally coupled to the grip crank 504. When a user presses one ormore of the grip crank 504, the follower arms 506 push a slider 508upwardly, which may trigger a sensor at the proximal end 530A togenerate a grip signal for controlling jaws of the corresponding endeffectors.

User console 110 may include a tracking system to track the positionsand orientations of UID 116. In some variations, the tracking system mayuse optical tracking, in which cameras may be mounted in front of theuser to capture continuous images of the UID movement. One or moremarkers can be printed or mounted on the UID 116 so that images can beprocessed to detect positions and orientations of the UID 116. Themarkers may be detectable in an image to determine a position ororientation of UID 116. In other variations, the camera can be mountedon the UID 116. Alternatively or in addition, the tracking system may bebased on electromagnetic tracking subsystem having an electromagneticsource. Magnetic tracking can determine the position/orientation ofmoveable sensors relative to a fixed transmitter within a definedoperating space. The sensors precisely measure the magnetic field from atransmitter, which is configured to generate a known set of fieldpatterns. The transmitted patterns are arranged such that the system canresolve a unique spatial position and orientation from the valuesmeasured by each sensor. For example, the user console 110 may have afield generator to generate a position-varying magnetic field toestablish a coordinate space in front of the user, while the UID 116 mayinclude one or more tracking sensors capable of measuring six degrees offreedom within the coordinate space. The tracking sensors may includecoils that respond to the electromagnetic field, e.g., inducing current.By measuring the coil current, a position and orientation of thetracking sensor and the UID 116 can be determined.

The tracked spatial state information can then be forwarded by the userconsole 110 to the control tower 130 as user input for robotic control.A control computer at the control tower 130 can process the user inputto generate control commands so that end effectors of the surgicalinstrument or tool closely follow the tracked movements of the UID 116,which includes translations and rotations along all three axes, i.e., 6DOF, in space. In addition to the spatial state, other gestures such asa squeeze or grip may also be tracked by one or more sensors, which canbe a proximity sensor detecting deformation or mechanical linkages beingpressed by user fingers. Once detected, the UID 116 may generate a gripsignal and transmit the grip signal to the user console. The signals canbe input to the control tower 130 for generating tool control commandssuch as grasping. For example, a grip signal may be translated into atool control command to close the jaws of a grasper, while a releasesignal may be mapped to an open jaw command. Furthermore, the gripsignal may command the surgical tool to apply grip force on tissues orsurgical devices.

The UID 116 may further include a finger clutch, such as touch sensorsor mechanical switches, to temporarily clutch the surgical robot. Whenclutched, the surgical robot 120 may pause the movement of one or moreof the tool or arm actuators in response to the clutch signal. Theteleoperation can be put on hold regardless of any UID movements, i.e.,spatial state and/or grip signals are ignored by the system. During theclutch period, the user can reposition the UIDs 116, for example, to thecenter of the tracking field, without causing unwanted movement ofsurgical robot or surgical tools. Similarly, the user may squeeze (orrelease) the UID housing 520 without changing the jaw angle or grippingforce of a grasper.

The user console 110 may further include sensors for tracking user eyesand/or head. In some variations, the eye tracking sensors may comprisecameras that take high frame rate images of user's eyes, projectors thatcreate a pattern of near-infrared light on user's eyes, and imageprocessing algorithms to analyze details of the user head or eyes andreflection patterns and to determine user's head position and gazepoint. The eye (or head) trackers can be mounted on the top or bottom ofthe open display 450, among other proper locations in front of the seatassembly 410 facing the user. In order to calibrate the eye (or head)trackers, spatial state information of the UID 116 and the surgicalrobot can be leveraged. For example, while the user is using the UIDs116 to manipulate an end effector of a surgical tool, a position of theend effector on the open display 450 can be derived as a reference gazepoint based on robot joint state and mapping between reference frames ofthe endoscope and the display 450. The tracker can then be calibrated bycomparing the reference gaze point and the tracked gaze point.

Tracking user's gaze point and/or head positions allow many applicationsfor robotic surgeries. For example, tracking user's head position mayhelp determine a spatial relationship between the head and the opendisplay that impacts 3D perception. The user console computer mayautomatically adjust seat and monitor position and tilt so that theuser's eyes are centered in front of the 3D display at a pre-determineddistance for best 3D perception. Furthermore, user's gaze can be used anindication of the user's engagement in teleoperation. For instance, gazepoint on the display can be used to automatically adjust the surgicalarea shown on the screen and/or endoscope position so that the focus ofthe user is at the center of the monitor where 3D perception is thebest. Gaze can be used to activate apps on a side panel if the usershift focus on the side panel. As another example, when it is detectedthat a user's gaze stays outside the open display over a period of time,i.e., the user is not observing the surgical site for some time, thesurgical robot can be paused or locked for patient's safety.

Graphic User Interface

Some exemplary aspects of a GUI for a robotic surgical system aredescribed herein. More variations of the exemplary GUI and integratedapps are described in detail in U.S. patent application Ser. No.15/842,485 titled “MULTI-PANEL GRAPHICAL USER INTERFACE FOR A ROBOTICSURGICAL SYSTEM” filed Dec. 14, 2017, which is incorporated herein inits entirety by this reference. In some variations, the GUI may bedisplayed in a multi-panel display at a user console that controls therobotic surgical system. Additionally or alternatively, the GUI may bedisplayed at one or more additional displays, such as at a control towerfor the robotic surgical system, at a patient bedside, etc. Anotherexample of a display on which the GUI may be present is an immersivedisplay such as those described in U.S. patent application Ser. No.15/724,185 titled “IMMERSIVE THREE-DIMENSIONAL DISPLAY FOR ROBOTICSURGERY” filed Oct. 3, 2017, which is incorporated herein in itsentirety by this reference.

Generally, the GUI for a robotic surgical system may provide informativeand/or interactive content, to thereby assist a user in performing asurgical procedure with one or more robotic instruments in the roboticsurgical system. In some variations, the GUI may include a multi-paneldisplay (or on multiple adjacent displays) on which content provided byvarious software apps may be overlaid or displayed proximate an image ofthe surgical site (e.g., from an endoscopic camera), such as during asurgical procedure. The software apps may be selectively arranged on themultiple panels to display their respective content in a reconfigurablemanner. Different layouts of the reconfigurable panels may result fromadjusting sizes and/or shapes of different panels. Additionally oralternatively, different layouts may result from the population ofdifferent content (e.g., different apps) in the multiple display panels.In some variations, a GUI may further display one or more tool widgetsconfigured to communicate information regarding surgical instruments ina convenient, efficient manner. For example, tool widgets may summarizehigh-priority information such as tool type, tool state, tool settings,and/or tool “lives” remaining (e.g., number of firings left in acartridge, etc.). Tool widgets may be overlaid over an endoscopic image,adjacent or proximate the endoscopic image, and/or or in any othersuitable portion of the displayed GUI.

FIG. 6A and FIG. 6B depict exemplary GUI variations 600A and 600B inwhich a video feed portion 610 of the real-time video app is displayedin a main panel between two sidebars on the display. Various apps can bedisplayed in the sidebars on either side of the main panel, where atleast some of the sidebar panels may be selectively hidden, minimized,made transparent, and/or pinned depending on the user's currentpreference. For example, in FIG. 6A, a stadium view app 611, a timer app612, a teleconferencing app 613, a procedure template/checklist app 614,a generator app 615, and an image viewer app 616, are displayed on thesidebars of the display. Similarly, in FIG. 6B, a control portion 610 aof the real-time video app, the timer app 612, the proceduretemplate/checklist app 614, and the image viewer app 616 are arranged invarious subpanels on sidebars around the main panel. Apps populating thesidebar panels may be depicted in a minimalistic style (e.g., simple andclean lines, etc.) or other suitable style. In addition oralternatively, the GUI may be configured to display specialized contentfrom one or more selected software apps in the main panel.

Other apps not depicted in the figures include a patient vitals app, atelestration app, a video labeling app, a video player app, a simulatorapp, an ergonomic setting app, among other variations of apps. A usermay access and activate an app from a tool bar or a quick access menu,(e.g., a quick access menu 620 in FIG. 6B), via an input device orinteraction with other suitable sensors. Apps may be represented bygraphical icons and/or titles in the tool bar or the quick access ring,and may be arranged in any suitable order, such as grouped by relevanceor functionality (e.g., relating to the surgical procedure, relating tothe patient, relating to team collaboration), alphabetically,user-selected “favorites,” most popular, etc. A user may also customizethe form factor of the tool bar or the quick access menu (e.g., circularor ring-shaped, rectangular or other grid of selectable icons ordescriptors, drop-down list, or any suitable format). The tool bar orthe quick access menu may be displayed in any suitable location on thedisplay (e.g., an open display in the user console). Furthermore, thequick access menu may be configured automatically based on previouseffective layouts of the quick access menu for certain kinds of surgicalprocedures, certain kinds of users, etc.

In some variations, the GUI may display a “tool kit” or tool barincluding one or more tool widgets showing surgical tool (instrument)status. For example, the tool kit shown in FIG. 6A includes tool widgets634A and 636A corresponding to a right-hand tool and a left-hand tool,respectively, and is displayed along a bottom edge of the main panel. Inother variations, the tool kit may additionally or alternatively bedisplayed in any suitable location. As shown in FIG. 6B, the tool kitmay include tool widgets 632, 634B, and 636B displayed on side bars ofthe display and corresponding to a first “backup” surgical instrumentlocated off-screen, a second surgical instrument controlled by aright-hand controller, and a third surgical instrument controlled by aleft-hand controller, respectively. Specifically, the tool widget 632,corresponding to the first “backup” surgical instrument, is displayed inan upper corner of a side bar, relatively out-of-sight. The tool widgets634B and 636B, corresponding to controlled second and third surgicalinstruments, are displayed in lower corners of right and left side bars,respectively.

In some variations, a tool widget may also display an alert or anotification in response to the detection of a trigger event occurringduring a workflow. For example, as shown in FIG. 6A, in the event thatone or more sensors (e.g., on a tool driver) detects that an instrumentis incorrectly loaded, an associated tool widget may display anotification 621 indicating the problem to a user. Additionally, aseparate notification 623 may be displayed elsewhere on the display. Theseparate notification 623 may, for example, include additional (e.g.,more detailed) information relating to the trigger event that is notdisplayed in the tool widget. Other examples of trigger events mayinclude exhaustion of “lives” of an instrument (e.g., out of apredetermined number of fires of a stapler), an instrument jamming, orother instrument malfunction. Any suitable trigger event for promptingan alert of a notification may be defined. In some variations, alerts ornotifications may be prioritized for display in order of urgency.

The image viewer app 616 may be in communication with a medical recordsdatabase or other suitable repository such that the image viewer app 616may receive medical images relating to the surgical procedure. Forexample, the image viewer app 616 may receive and display pre-operativeimages (e.g., X-ray, CT, MRI, ultrasound, etc.) of the patient. Suchdisplay of pre-operative images may allow a surgeon and/or other user(s)to easily view pre-operative images before, during, and/or after asurgical procedure and may help the surgical team make better, moreinformed decisions relating to the surgical procedure. For example,pre-operative images may be displayed via the image viewer app 616 inorder to facilitate pre-operative planning, such as the surgical teamreviewing a surgical plan at the outset of a case. As another example,pre-operative images may be displayed via the image viewer app 616side-by-side with real-time, intra-operative images obtained with anendoscopic (e.g., to assess margins of a tumor to be excised).

The real-time video app is configured to receive and display one or morereal-time image data from devices capturing images of a surgicalworksite during a surgical procedure. In some variations, the real-timevideo app is configured to receive and display information in additionto an endoscopic video feed from the robotic surgical system, such asthrough additional (e.g., third-party) endoscopic devices, ultrasoundmachines, etc. Display of additional real-time data streams to a usermay, for example, help enable surgical staff to view more information(e.g., from different angles or perspectives, with different imagingaids such as ICG or other imaging agents, etc.) that may help them makebetter treatment decisions. In some variations, the real-time video appmay additionally or alternatively receive image data from an endoscopicvideo feed from the robotic surgical system.

The patient vitals app may be in communication with one or more sensorstracking patient vital signs (or in communication with a memory devicestoring the same) such as pulse, blood pressure, oximetry data,respiratory rate, temperature, and the like. The display of patientvitals on the display may provide a surgeon and/or other user with easyaccess to a status of the patient (e.g., without having to ask a presentanesthesiologist). The display of patient vital signs in the patientvitals app may, for example, help enable a surgeon react more quickly toemergency situations. Furthermore, the patient vitals app may provide avisual and/or audio alert for trigger events, such as a patient vitalmeeting a predetermined threshold value (e.g., heart rate exceeding apredetermined value).

The procedure template app 614 may be in communication with a proceduredatabase stored in memory, such that it may receive data relating toprocedure planning. The procedure template app 614 may generate a listof items relating to performance of a surgical procedure. For example,the procedure template app 614 may display a checklist of surgical tasksthat are part of a surgical procedure, list of equipment or tools neededfor the surgical procedure, list of operating room setup tasks, aschematic diagram of port location and arias/surgical instrument setup,etc. In some variations, a checklist may be a template list or may becustomized (e.g., a template checklist that has been fine-tuned oradjusted for a particular patient, or an otherwise customized list). Theprocedure template app 614 may, for example, provide a way for thesurgical team to view procedure steps, equipment, and/or setup tasksbefore or at the outset of a surgical procedure.

The timer app 612 may, for example, track duration of the surgicalprocedure and/or duration of segments of the surgical procedure (e.g.,individual surgical tasks and other tasks performed, and/or groupsthereof). In some variations, the timer app 612 may provide a way formedical staff to easily monitor progress of the surgical procedureintraoperatively. Additionally or alternatively, the timer app 612 mayanalyze (or facilitate analysis of) performance of the surgicalprocedure post-operatively to help enable the surgical team to identifypossible ways to improve efficiency, communication, etc. In somevariations, data gathered via the timer app 612 may be displayed onother displays (e.g., additional displays in the operating room) and/orcommunicated and stored for later analysis. For example, data gatheredvia the timer app 612 may be uploaded to a web portal or database tohelp enable an intra-operative and/or post-operative review of thesurgical procedure.

The stadium view app 611 provides a real-time view of the roboticsystem, patient table or bed, and/or staff in an operating room during aprocedure. The stadium view app 611 may, in some variations, receivereal-time or near real-time information relating to a current positionof the robotic arms, patient table, and/or staff and the like, generatea rendering (graphical representation) of the operating room environmentbased on the received information, and display the rendering to theuser. In some variations, the rendering may be in 3D, but mayalternatively be in 2D. Alternatively, the rendering may be generated bya remote device (e.g., a separate processor) and passed to the stadiumview app 611 for display. Accordingly, the displayed rendering mayprovide the user with an “outside-the-patient-body” view of the roboticsurgical system, the patient, and/or staff, etc. in the operating room.The user may, for example, monitor status of the robotic system such astool status, potential collisions, etc., and communicate to othermembers of the surgical team about such status and resolution of anyissue.

The teleconferencing app 613 may enable a user to contact a colleague orother contact before, during, and/or after a surgical procedure. Forexample, the teleconferencing app 613 may enable communication over acellular network, a wired or wireless internet network (e.g., overWiFi), a direct line network connection, or in any suitable manner. Insome variations, the teleconferencing app 613 may store contactinformation including but not limited to name, picture, role or title,location, phone number or other contact, and the like. Through theteleconference app 613, a user may, for example, seek consultation witha contact for advice or other telementoring, or seek any other suitablekind of collaboration for a surgical procedure. The teleconferencing app613 may facilitate audio and/or visual collaboration, such as withtelephone and/or video conferencing, and/or screen sharing.

The telestration app may enable one or more users to annotate adisplayed image or other aspect of the GUI. For example, a telestrationapp may display a palette of one or more annotation tools. Theannotation tools may be used to mark up or label a displayed image suchas an endoscopic image, and the annotated image may then be sharedbetween collaborators (e.g., among different GUIs simultaneouslydisplayed on different displays), saved for reference or futureanalysis, etc. For example, an annotated image may be used to moreclearly communicate with a collaborator the location of lesion margins,nearby lymph nodes, and/or other critical anatomical structures (e.g.,anatomical targets, tissue to avoid), etc. Collaborators may be amongthe same surgical team or in the same operating room, and/or may beexternal to the operating room (e.g., remote collaborators, such as ateleconferencing mentor).

In some variations, a video recording of a surgical procedure may beobtained, such as throughout a surgical procedure. The video labelingapp may include annotation tools (e.g., similar to those described abovefor the telestration app) that may be used to annotate or otherwiselabel the recorded surgical procedure videos. For example, the videolabeling app may help enable users to associate a surgical procedurevideo with a particular patient (e.g., annotate with patient name,medical record number, etc.), in order to enable future access to thevideo such as for post-operative review.

The video player app may, for example, be in communication with a videodatabase such that the video player app may receive a video (or apointer to a video) and display it on the GUI. The video player app maydisplay, for example, an instructional or training video for a relevantsurgical procedure or surgical task, or for other tasks relating to thesurgical procedure (e.g., setup of ports for docking the robotic arms).In some variations, the video player app may be used by users to reviewvideos in a pre-operative setting, such as to prepare for a surgicalprocedure. Additionally or alternatively, the video player app may beused to review videos in an intra-operative setting, such as to helpresolve a complication that has arisen during the surgical procedure.However, other suitable videos may be played. Furthermore, it should beunderstood that variants of a video player app (e.g., a music player)may be provided via the GUI.

The generator app 615 may enable control of one or more settings of asurgical instrument. For example, a GUI including a generator app 615 ata user console may enable a surgeon sitting at the user console tocontrol settings of a surgical instrument directly. In some situations,this may increase overall efficiency of the surgical procedure, as asurgeon at the user console may avoid having to ask another member ofthe surgical staff to change settings.

Control Tower

FIG. 7 is a schematic diagram illustrating an exemplary control tower130 for the surgical robotic system, in accordance with aspects of thesubject technology. The control tower 130 may house a team display 702,a nurse display 706, accessory shelves 704 and 708, central computers710 including at least a visualization computer, a control computer andan auxiliary computer, a storage space 712 for cable management and CO₂tanks and casters 714. The team display 702 can provide a view of thesurgical site from the patient-side and a set of controls for endoscopeand video configurations. For example, the team display 702 mayduplicate the entire surgical image seen on the user console opendisplay 450, including endoscope video and apps. In addition toproviding instrument status indicators (ready vs. not in use),instrument and foot pedal activation status indicators (activation vs.no activation), and endoscope status/horizontal indicators, the teamdisplay may also provide additional on-screen icons, off-screenindicators when instrument is out of view, warnings, alerts, systemstatus, menu, setting and options, and other relevant information, asappropriate.

The nurse display 706 may be a smaller display house on the controltower 130 and may be used by surgical staff to interact with the systemin addition to the team display 702. The nurse display may show thestatus of the endoscope, illuminator, and generator, and allowadjustment to the settings of these devices. In addition, the nursedisplay 706 may allow user login and access to surgical data andsoftware apps. As shown in FIG. 7 , above and below the nurse display706 are two accessory shelves 704 and 708, though other arrangements ofthe relative locations between the accessory shelves 704, 708 and thenurse display 706 may be possible. In some variations, the two shelves704, 708 allow for placement of accessory equipment, such asinsufflators, an advanced light engine (ALE), and an external,non-integrated electrosurgical unit (ESU). The ESUs may includemonopolar, bipolar, advanced bipolar, and ultrasonic modalities. The ALEcontains a high-intensity light source to illuminate the surgical siteand electronics for initial processing of endoscopic video images. Anendoscope cable may be connected directly from the ALE to the endoscopeto provide communication and illumination.

The central computers 710 may include a visualization computer, acontrol computer, and an auxiliary computer. The visualization computercomprises one or more processors mainly for advanced processing of thevideo image received from the endoscope via the advanced light engine(ALE) and forwarding the processed video to various displays. Thevisualization computer may support the graphic user interface,touchscreen input at nurse display and team display, apps responsiblefor real-time and/or low-latency endoscope video or other video inputsto the system, apps responsible for providing the user time- orsafety-critical information via a user console or team display,communication channel between visualization computer and the surgicalrobotic system, communication of the ALE settings information,activation/deactivation of external ESUs, and audio mixing and export ofthe audio/video for archival.

The control computer receives user input from the user interface devicesat the user console as well as status information from the surgicalrobot (e.g., table, arms, and instruments), and computes the controlcommands based on the user input and the robot status. The controlcommands can then be dispatched to the surgical robot to performrobotically assisted surgeries. The control computer may support therobotic control of the surgical robotic system, including teleoperation,translating UID movement into robotic motion, receiving and transmittingreal time data and non-real time data, exporting system data for loggingand storage, facilitating table control, arm control and tool control,and managing communication channels between all central computers andthe surgical robotic system. The auxiliary computer may also recordsystem logs to internal storage, export system data to internal orexternal components, and function as a backup computer to thevisualization computer and the control computer.

Other accessories (not shown in FIG. 7 ) of the control tower 130 mayinclude speakers and microphones together with volume controls, and/oraudio jacks for audio input and output assisting vocal communicationbetween the surgeon at the user console and bedside staff. The controltower may also include an uninterruptible power supply (UPS) thatprovides power in the absence of mains power. The control tower can bemobile for optimal positioning in the operating room when the casters714 (electronic and/or manual) are unlocked. When not in transport mode,the wheel locks/braking system is capable of preventing unwantedmovements.

System Architecture

FIG. 8 is a block diagram illustrating an exemplary hardware componentsof a surgical robotic system 800, in accordance with aspects of thesubject technology. The exemplary surgical robotic system 800 mayinclude a user console 110, a surgical robot 120, and a control tower130. The surgical robotic system 800 may include other or additionalhardware components; thus, the diagram is provided by way of example andnot a limitation to the system architecture.

As described above, the user console 110 comprises console computers811, one or more UIDs 812, console actuators 813, displays 814, a UIDtracker 815, foot pedals 816, and a network interface 818. A user orsurgeon sitting at the user console 110 can adjust ergonomic settings ofthe user console 110 manually, or the settings can be automaticallyadjusted according to user profile or preference. The manual andautomatic adjustments may be achieved through driving the consoleactuators 813 based on user input or stored configurations by theconsole computers 811. The user may perform robot-assisted surgeries bycontrolling the surgical robot 120 using two master UIDs 812 and footpedals 816. Positions and orientations of the UIDs 812 are continuouslytracked by the UID tracker 815, and status changes are recorded by theconsole computers 811 as user input and dispatched to the control tower130 via the network interface 818. Real-time surgical video of patientanatomy, instrumentation, and relevant software apps can be presented tothe user on the high resolution 3D displays 814 including open orimmersive displays.

Unlike other existing surgical robotic systems, the user console 110disclosed herein may be communicatively coupled to the control tower 130over a single fiber optic cable. The user console also providesadditional features for improved ergonomics. For example, both an openand immersive display are offered compared to only an immersive display.Furthermore, a highly-adjustable seat for surgeons and master UIDstracked through electromagnetic or optical trackers are included at theuser console 110 for improved ergonomics. To improve safety, eyetracking, head tracking, and/or seat swivel tracking can be implementedto prevent accidental tool motion, for example, by pausing or lockingteleoperation when the user's gaze is not engaged in the surgical siteon the open display for over a predetermined period of time.

The control tower 130 can be a mobile point-of-care cart housingtouchscreen displays, computers that control the surgeon'srobotically-assisted manipulation of instruments, safety systems,graphical user interface (GUI), light source, and video and graphicscomputers. As shown in FIG. 8 , the control tower 130 may comprisecentral computers 831 including at least a visualization computer, acontrol computer, and an auxiliary computer, various displays 833including a team display and a nurse display, and a network interface838 coupling the control tower 130 to both the user console 110 and thesurgical robot 120. The control tower 130 may also house third-partydevices, such as an advanced light engine 832, an electrosurgicalgenerator unit (ESU) 834, and insufflator and CO₂ tanks 835. The controltower 130 may offer additional features for user convenience, such asthe nurse display touchscreen, soft power and E-hold buttons,user-facing USB for video and still images, and electronic castercontrol interface. The auxiliary computer may also run a real-timeLinux, providing logging/monitoring and interacting with cloud-based webservices.

The surgical robot 120 comprises an articulated operating table 824 witha plurality of integrated arms 822 that can be positioned over thetarget patient anatomy. A suite of compatible tools 823 can be attachedto or detached from the distal ends of the arms 822, enabling thesurgeon to perform various surgical procedures. The surgical robot 120may also comprise control interface 825 for manual control of the arms822, table 824, and tools 823. The control interface can include itemssuch as, but not limited to, remote controls, buttons, panels, andtouchscreens. Other accessories such as trocars (sleeves, sealcartridge, and obturators) and drapes may also be needed to performprocedures with the system. In some variations, the plurality of thearms 822 includes four arms mounted on both sides of the operating table824, with two arms on each side. For certain surgical procedures, an armmounted on one side of the table can be positioned on the other side ofthe table by stretching out and crossing over under the table and armsmounted on the other side, resulting in a total of three arms positionedon the same side of the table 824. The surgical tool can also comprisetable computers 821 and a network interface 828, which can place thesurgical robot 120 in communication with the control tower 130.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that specificdetails are not required in order to practice the invention. Thus, theforegoing descriptions of specific embodiments of the invention arepresented for purposes of illustration and description. They are notintended to be exhaustive or to limit the invention to the precise formsdisclosed; obviously, many modifications and variations are possible inview of the above teachings. The embodiments were chosen and describedin order to best explain the principles of the invention and itspractical applications. They thereby enable others skilled in the art tobest utilize the invention and various embodiments with variousmodifications as are suited to the particular use contemplated. It isintended that the following claims and their equivalents define thescope of the invention.

The methods, devices, processing, and logic described above may beimplemented in many different ways and in many different combinations ofhardware and software. The controllers and estimators may compriseelectronic circuitry. For example, all or parts of the implementationsmay be circuitry that includes an instruction processor, such as aCentral Processing Unit (CPU), microcontroller, or a microprocessor; anApplication Specific Integrated Circuit (ASIC), Programmable LogicDevice (PLD), or Field Programmable Gate Array (FPGA); or circuitry thatincludes discrete logic or other circuit components, including analogcircuit components, digital circuit components or both; or anycombination thereof. The circuitry may include discrete interconnectedhardware components and/or may be combined on a single integratedcircuit die, distributed among multiple integrated circuit dies, orimplemented in a Multiple Chip Module (MCM) of multiple integratedcircuit dies in a common package, as examples.

The circuitry may further include or access instructions for executionby the circuitry. The instructions may be stored in a tangible storagemedium that is other than a transitory signal, such as a flash memory, aRandom Access Memory (RAM), a Read Only Memory (ROM), an ErasableProgrammable Read Only Memory (EPROM); or on a magnetic or optical disc,such as a Compact Disc Read Only Memory (CDROM), Hard Disk Drive (HDD),or other magnetic or optical disk; or in or on another machine-readablemedium. A product, such as a computer program product, may include astorage medium and instructions stored in or on the medium, and theinstructions when executed by the circuitry in a device may cause thedevice to implement any of the processing described above or illustratedin the drawings.

The implementations may be distributed as circuitry among multiplesystem components, such as among multiple processors and memories,optionally including multiple distributed processing systems.Parameters, databases, and other data structures may be separatelystored and managed, may be incorporated into a single memory ordatabase, may be logically and physically organized in many differentways, and may be implemented in many different ways, including as datastructures such as linked lists, hash tables, arrays, records, objects,or implicit storage mechanisms. Programs may be parts (e.g.,subroutines) of a single program, separate programs, distributed acrossseveral memories and processors, or implemented in many different ways,such as in a library, such as a shared library (e.g., a Dynamic LinkLibrary (DLL)). The DLL, for example, may store instructions thatperform any of the processing described above or illustrated in thedrawings, when executed by the circuitry.

Also, the various controllers discussed herein can take the form ofprocessing circuitry, a microprocessor or processor, and acomputer-readable medium that stores computer-readable program code(e.g., firmware) executable by the (micro)processor, logic gates,switches, an application specific integrated circuit (ASIC), aprogrammable logic controller, and an embedded microcontroller, forexample. The controller can be configured with hardware and/or firmwareto perform the various functions described below and shown in the flowdiagrams. Also, some of the components shown as being internal to thecontroller can also be stored external to the controller, and othercomponents can be used.

The invention claimed is:
 1. A surgical robotic system, comprising: anoperating table having a table top mounted on a pedestal; a tableadapter directly connected to the pedestal; a plurality of robotic armsdirectly connected to the table adapter of the operating table; aplurality of robotic surgical instruments, each coupled to a distal endof one of the plurality of robotic arms; a user console having one ormore user interface devices and a sensor for tracking a gaze point of auser; and a control tower having a control computer communicativelycoupled to the one or more user interface devices and the plurality ofrobotic surgical instruments, wherein the control computer is configuredto: receive user input from the one or more user interface devices atthe user console as the user manipulates the one or more user interfacedevices; and generate control commands for the plurality of roboticsurgical instruments to follow the user input; automatically adjust, atan open display, at least one endoscopic image of a surgical site to theuser according to the gaze point of the user; and pause or lock at leastone of the robotic surgical instruments when the gaze point of the userdeviates for a predetermined time period.
 2. The system of claim 1,wherein the table top includes a head section, a torso section, and aleg section along a longitude axis of the table top.
 3. The system ofclaim 2, wherein the plurality of robotic arms comprises four roboticarms, wherein two of the robotic arms are configured to be stowed underthe head section of the table top, and wherein another two of therobotic arms are configured to be stowed under the leg section of thetable top.
 4. The system of claim 2, wherein the plurality of roboticarms comprises four robotic arms, wherein two of the plurality ofrobotic arms are coupled to the pedestal on each side of the longitudeaxis of the table top.
 5. The system of claim 4, wherein each of the twoarms coupled to one side of the table top is configured to be pivotedoutwardly and extended upwardly relative to the table top on the sameside for a first surgical procedure.
 6. The system of claim 4, whereinone of the two arms positioned on one side of the table top isconfigured to be pivoted under the table top across the longitude axisand extended to the other side of the table top for a second surgicalprocedure.
 7. The system of claim 4, wherein the table top is configuredto translate and rotate about the longitude axis, and wherein the fourrobotic arms are configured to maintain poses relative to the table topduring translation and rotation of the table top.
 8. The system of claim1, wherein the one or more user interface devices are ungroundedhandheld user input devices.
 9. The system of claim 8, wherein the userconsole further comprises a tracking system for tracking positions andorientations of the one or more user interface devices.
 10. The systemof claim 1, wherein the user console comprises an ergonomic seat. 11.The system of claim 1, wherein the open display includes a plurality ofuser console configurations include a seated configuration, a reclinedconfiguration, and an elevated configuration.
 12. The system of claim10, wherein the one or more user interface devices is further configuredto receive a user input indicating a selection of at least one softwareapplication relating to the robotic surgical system, and wherein theopen display is further configured to render content from the at leastone selected software application.
 13. The system of claim 10, whereinthe user console further comprises an immersive display.
 14. The systemof claim 10, wherein the control tower further comprises a visualizationcomputer communicatively coupled to the open display and an endoscopemounted on one of four robotic arms.
 15. A surgical robotic system,comprising: an operating table mounted on a pedestal; a table adapterdirectly connected to the pedestal; a plurality of robotic arms directlyconnected to the table adapter of the operating table; a user consolehaving one or more user interface devices and a sensor for tracking agaze point of a user; and a controller coupled to the one or more userinterface devices, wherein the controller is configured to perform:receiving user input from one or more user interface devices at a userconsole of a surgical robotic system as the user manipulates the one ormore user interface devices; generating control commands for a pluralityof robotic surgical instruments of the surgical robotic system to followthe user input; automatically adjusting at least one endoscopic image ofa surgical site to the user according to the gaze point of the user; anddisabling at least one of the robotic surgical instruments when the gazepoint of the user deviates for a predetermined time period.
 16. Thesystem of claim 15, wherein the one or more user interface devices areungrounded handheld user input devices.
 17. The system of claim 15,wherein the one or more user interface devices is further configured toreceive a user input indicating a selection of at least one softwareapplication relating to the robotic surgical system.
 18. A surgicalrobotic system, comprising: an operating table having a table topmounted on a pedestal; a table adapter directly connected to thepedestal; a plurality of robotic arms directly connected to the tableadapter of the operating table; a plurality of robotic surgicalinstruments, each coupled to a distal end of one of the plurality ofrobotic arms; a user console having one or more user interface devicesand a sensor for tracking a gaze point of a user, wherein at least oneendoscopic image is adjusted according to the gaze point of the user;means for receiving user input from the one or more user interfacedevices at the user console as the user manipulates the one or more userinterface devices; means for generating control commands for theplurality of robotic surgical instruments to follow the user input,wherein at least one of the robotic surgical instruments is disabledwhen the gaze point of the user deviates for a predetermined timeperiod.
 19. The system of claim 18, further comprising a visualizationcomputer communicatively coupled to an open display and an endoscopemounted on one of the plurality of robotic arms.